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iBSTHRCT 

A model of an evaluation process for a college level 
aatheaatics course using Banaaeaent procedures is described. It is 
suggested that by design ino the evaluation process as part of a 
oanageaeat procedure, data useful for making policy decisions will be 
generated and the success of the instructional process will be 
insured. The foundation of the systei is the developient of a 
procedures manual that generates questions about policies that affect 
the adainistration, the students, and the operation and effectiveness 
of the staff and^he proaram. The second phase of evaluation involves 
data analysis conb«^ing quej^tions of staff size, program structure, 
attendance requiremelNks, orading procedures, and prcgraa costs. Data 
collection, the third phase, provides a record of student progress 
and weekly cumulative reports- Quarterly reviews and tutor 
evaluations are also part of the procedure used. The final phaise of 
evaluation is dissemination of reports to the staff, adiinistrati ve 
units, and agencies' outside the university. The reports range froa 
one-page suamaries to full-scale reports docusenting procedures and 
student achieveaent. It is suggested that the aanageaent approach to 
evaluation evolved because the success of this prograa for 
aatheaatics depended upon efficient program aanageaent as well as 
quality instruction and naterials. The evaluaticn ccaponerts of the 
Bodel are viewed as having a large effect on the success of these 
prograas- CAuthor/SF> 
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INTRODUCTION 

Educators have "initiated many types of curricu- 
lar approaches. However, these alternative instructional 
strategies tend to be expensive. Current demands for 
cost effectiveness necessitate the integration of man- \ 
agement techniques and evaluation data in order to make 
policy decisions. This paper describes a model of an 
evaluation process using management procedures. 

The components of the model include documentation, 
revision, and dissemination. These procedures can be 
adapted to a variety of programs, and they provide not 
only evaluation data but also a management strategy. 
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33ESCRIPTION OF THE PROGRAM 

A pre-calculus math program was developed as an 
alternative instructional approach for university stu- 
dents entering academic programs which require calcu- 
lus. It was specifi-cally viesigried for students who 
had not received adequate preparation in algebra and 

trigonometry during high school, or those who did not 

< 

have the confidence in their skills to enroll in calculus. 

This program was piloted in 197^ with 35 students. 
The administrative effort was directed totally toward 
the development of the programmed materials. lack 
of administrative procedures resulted in an extremely 
high dropout rate and high numbers of incompletes which 
were often never resolved. Few records were kept on 
student progress or program evaluation. 

Initially, program policies and procedures developed 
primarily as a result of crisis situations arising from 
the nature of the program. After several quarters it 
became apparent that program changes as well as syste- 
matic management procedures and policies were required 
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to control the dynamic process involved in a self- 
paced program of instruction. 

The first step in organizing the program was to 
structure the students' study time. A new procedure v/as 
initiated which "required each student who enrolled in the 
alternative program to register in a class section. 
During class timp, students studied their programmed 
materials seeking help from tutors when necessary. 

<j. The second major step in structuring the program 
was the institution of deadlines for material covered. 
The student could work at his own pace as long as he met 
che minimum time frame agreed upon by the administrators 
of the program. ContincHis testing was made available 
in a math lab, which was also manned by tutors. 

With the increasing size of the program, more 
efficient management procedures were essential. As a 
result of these management policies fewer problems arose. 
The efficiency of the program was the result of sever- 
al interdependent factors « 



1. A graduate assistant was hired to work 20 
hours per week to cooi-dinate the tutors ajid 
to administer the policies and procedures. 

Zi A detailed procedures manual was developed. 

3. A formalized data collection method was instituted. 

4. Data analysis was dene on various aspects of 
the program. 

5« A detailed program evaluation was developed 
as a result of management-generated data. 

6. A formalized reporting system of the program 
evaluation was begun. 

The remainder of the paper sets forth a description 
of the evaluation components of the model and their 
effects on the success of the program. The documentatio.i 
stage includes the development of the procedures manual 
and the collection of data. The revision component in- 
cludes data analysis and assessment of current policies. 
The dissemination phase includes all weekly and quarterly 
reports used to offer documentation of program success. 



MANAGENENT PROCESS 

I. Documentation! 

The Procedures Manual 
During the expansion of the math program various 
policies, procedures, and rules were developed, both 
formally and informally, to help alleviate specific 
problems. These procedures served to provide some con- 
sistency in the tutoring, testing, and grading proce- 
dures. It became necessary to put all the policies and 
procedures together so periodic evaluations could be per- 
formed, materials could be readily available to orient 
new staff, and a formal description of the program would 
exist. A procedures manual was developed during spring 
quarter, 1979 » to document these opera;fcions. The manual 
describes the administrative, testing an-d tutoring ' ' 
procedures, and the .staffing requirements. It also 
contains copies of all forms used in the operation of 

the program as well as handout materials prepared for 
«« 

students. 

See Appendix-Procedures Manual Table of Contents 



Because the program continually strives to im- 
prove its procedures, which in turn increases student 
success, changes in policies and/or procedures often 
occur which require documentation. The procedures man- 

* 

ual is used as a guideline for evaluation. It helps 
generate questions about policy i Are they needed? 
Are they effective? Should they be changed? It also 
generates questions about the effecj;iveness and ade- 
quate availability of staffs Are there enough staff 
to cover the math lab? Are. the tutors effective with 
the students? Do the tutors have the proper skills to 
be successful as a tutor? 

Using the procedures manual as a guide, more re- 
liable and effective decision-making policies are pos- 
sible. However, competent decision-making can only re- 
sult from an appraisal of all pertinent informatipn 
aiid data around which the decision must be made. There 
fore, systematic collection of information and data is 
essential in producing capable decisions. 



Data Collection P.roGess 

Student Data « Data on student retention are obtained 
from daily attendance records and weekly progress re- 
ports. The weekly.- student progress reports are com- 
puter printouts indicating students* weekly testing 
results. 

At the end of each quarter data are collected and 
evaluated on the students who dropped out of the pro- 
gram, when they dropped, and their progress before they 
dropped. 

Final grade sheets are used to determine the num- 
ber of students who coinpleted the program. Grade dis- 
tributionc ore reported. Incomplete grades are then 
followed-up through the next quarter until they are 
resolved. 

Data on the success of students completing the pro- 
gram before entering calculus and their success in a 
calculus series is gathered and analyzed in an independent 
study. 
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Program Datai Data are gathered throughout the quarter 
on facility and material utilization by use of a lab 
request card. These cards are completed each time a test 
is administered or- a stuSent utilizes audio-visual 

materials. At 1;he" end of each quarter, the data from the 

«■>• 

cards are used to quarrfiiV information to generate re\ 
ports indicating the number of tests administered each day 
the math lab is open. J^om this data it is possible to 

estimate the space needed and staff required. The audio- 

V 

visual d^ta is used th make decisions on materials required, 

the number of tape copies needed for projected student 

'\ V 

\, 

enrollment,^ and the staff requirements (lab assistants) 

to operate the audio- visual equipment. 

\ 

Data wh^ch are maintained on grade distribution are 

\ 
\ 

used to e validate student success. 

Data are collected to ascertain student satisfaction 




with the program^ Students are asked to complete a 
student satisfacti^a^ questionnaire at the end of each quarter. 
The students rate tlje course content, tutor effective- 
ness, and presentatiion as well as offer suggestions and 

\ 
I 

comments. f 



staff Data i ^y/pa.st enrollment and lab utilization data 
are used to predict staff needs for projected enrollment. 
The maximum number of students that can be accomodated 

is computed. « 

Evaluation data on tutor performance^ are collected 
throughout the "garter. Tutors are required to maintain 
testing and attendance records on the tutored students. 
These records are evaluated weekly by the tutor coordin- 
ator for neatness and accuracy. Written memos are kept 
on tutors' punctuality and attendance at tutor sessions 
and tutor meetings. At least once per quarter the tutor 
coordinator prepares a written observation of each tutor 
in a tutoring session. The lab supe^^dsor furnishes both 
written and oral reports of tutor performance throughout 
the quarter. The tutors are assessed in such areas as 
punctuality, initiative, cooperativeness, and job know- 
ledge. As a result of these various suroative evaluation 
techniques the tutor coordinator prepares a student tutor 
evaluation which is used in the decision for continued 



employment. The outcome of such an evaluation process 
/^are highly competent and successful tutors which con- 

i \ 

I l^ribute to a successful program. 

II. Revision I 

Questions Answered by Data Analysis 

As a result of the analysis of the collected data, 
small scale studies are organized to evaluate the 
necessity and effectiveness of certain policies. 

A study was developed and conducted over a two- 
quax'ter period on the relationship between success in the 
course and attendance in class. The results were 
us-ed as a criteria to evaluate whether or not attendance 
should be strongly recommended to students. 

A study of the incomplete grades was used to 
evaluate the appropriateness of procedures controlling 
testing deadlines. Answers to such questions as i "Was 
the minimiim testing schedule realistic?* can be answered 
by evaluating the incomplete grades and drops. 

Data analysis of the math lab v4.deo tape utilization 
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and testing over a period of four quarters are being 
used to answer questions regarding the appropriateness 
of the testing facility. 

An independent follow-up study was performed to 
discover how successful the students completing the 
alternative program were in the first sequence in calculus. 
This information was used to revise the programmed 
materials . 

Evaluation Process 

With the development of the procedures manual and 
Liore systematic data collection methods, evaluation 
of policies and procedures in the program became more 
concrete. U^ing the procedures manual as a criteria 
of evaluation each aspect of the program is assessed 
quarterly. 

The written goals of the program are evaluated ac- 
cording -^,0 the measureable objectives built into the 
system. Changes are made to satisfy the objectives and 
stated goals. 




Using collected data the math lab facility is 
appraised in terms of its location, ':!ize, and availability 
to students. Final decisions are made on the next 
quarter's schedule. 

The tutorial services are evaluated in areas of total 
number of tutors needed, availability to students, exper- 
tise in content areas, expertise in interpersonal re- 
lationships, dependability, and responsibility. As a re- 
sult of this evaluation, a tutor training program has been 
developed and implemented to offer tutors understanding 
and expertise in the tutoring process. 

Testing procedures are assessed on the availability 
of the testing facility, types of test items and their 

^ 

appropriateness, number of tests needed, the effectiveness 
of test administration, and problems arising from com- 

s 

puter generated tests. Those procedures which are not 
usef\il are abandoned. Any new procedure which allows a 
more effective testing process is accepted and tested. 
Using utilization data and the procedures manual, . 
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support materials such as audio and video tapes and 
books are assessed in the areas of availability and staff re- 
quireinents. Decisions are made for the following quarter and 
steps are taken to satisfy the projected needs. 

Job descriptions listed in the procedures manual of 
all staff employed in the program are evaluated in terms of 
stated duties and responsibilities. Each of the job 
descriptions is appraised as to whether the written pro- 
cedures and actual practice coincide. Where discrepancies 
occur, a decision is made to correct either the manual or 
the practice. As a result, the program has the best utiliza- 
tion of staff hours. 

Grading policies and procedures are evaluatsd on tl^ir 
fairness and appropriateness for such a program, and 
whether these policies are conducive to the total leam- 
ing process. Changes are made which will allow the student 
a greater opportunity to gain the knowledge he needs for 
success in calculus. One such example is the recent change 
in policies governing incomplete grades. The former policy 



set no time limit for the resolution of incomplete grades. 
It was. found that many of the grades were never resolved. 
A d;ecision was made to set a deadline for the follo|^g 
quarter after receiving the "I". If not resolved by that 
deadline, the "I" Would become an "E". As a result of the 
policy change, more "I" grades are . resolved. 

Students who choose the alternative program must 
have a comprehensive orientation to the program require- 
ments- -and procedures. -The procedural aet^iods -of -in^eetrina- 
ting students in self-paced programmed learning are continually 
evaluated. The quarterly course student guide is the 
r.ajor source of information for the students. This guide' 
contains all the information the students need to know 
about course expectations, course content, testing pro- 
cedures, grading proceduires, and testing, deadlines. At 
the end of each quarter, this student guide and the pro- 
cedures developed to implement it are evaluated on their 
effectiveness and are changed as needed. 

The evaluation process is an ongoing method of assessing 



the success of the program in terras of efficiency of 
staff hours as well as success of the students iti the 
program. 

III. Disseminations 

■ 

Generation of Reports 

The final phase of evaluation is the generation of 
reports- for dissemination to the staff, administrative 
units, and agencies outside the university. 

Attendance and testing reports enable the math 
faculty coordinator to disQ^iss each student's situation 
from a knowledgeable position during student conferences. 
The tutor coordinator compiles weekly attendance records 
and computer generated reports of students' weekly pro- 
gress which is given to the math faculty coordinator 
for her use in student conferences. 

A formalized in-house report is written each quarter 
to document the quarterly business. At times it is 
beneficial to send this report to individuals outside 
the program to inform them of its success. 
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* 

This report contains several distinct sections i 

1 . There is a detailed description of staff accom- 
plishments, staff changes, and the total number of 
staff ho\irs employed during the quarter. 

2. Test utilization data not only numerically but 

also graphically depicts daily use, and a comparison chart 
of the present and previous quarter demonstrates any changes. 
3- A materials utilization report shows various data 
using math lab materials such as audio-visual tapes 
and books, 

^. A comparison of the initial enrollment of the quarter 
to the final enrollment figures and data on the 
students who dropped the course are incorporated. 
Grade distribution as well as a description of resolu- 
tion of incomplete grades of the previous quarter are 
evaluated. • . " 

5. The final section of the quarterly report is a' re- 
view and analysis of the student satisfaction questionnaires. 
Answers .to such questions. Why are the students enroUed 
in the alternative program?, How many iiours per day are 

o IS 
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the student spending on the course?, Did they thinK 
the grading was fair?, Did the students like the 
programmed materials?, and Did they like the tutors?, 
are presented. Also included are some general comments 

■ 

to which the student responded. 

These reports serve to offer measurea"ble evaluation of 
program data. The flexibility of the reporting format 
allows sections of existing reports to be used to send to 
appropriate administrators outside the program. It allows a 
formal description to be available to people who inquire 
about the program. The documentation process inherent 
in the program provides a history which can be used in 
planning. 

SUMMARY 

The management approach to instructional evaluation 
evolved because the success of this non-traditional 
self-paced program for mathematics depended upon efficient pro 
gram management. Good management depends on data which are 
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easily accessible and quantifiable. Therefore » data 
collection and analysis becomes part of the management 
process. 

For the management process to be effective, the 
administrators or managers of the program must also be 
evaluators of the policies and procedures which govern 
the program. As a result of the data collection and analysis 
the administrators can integrate evaluation procedures 
within the mangement process. 
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